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METHOD OF USING HUE TO INTERPOLATE COLOR PIXEL SIGNALS 

RELATED APPLICATION 

5 This patent application is a continuation-in-part of US Patent Application Serial No. 

09/410,800, titled "Method of Interpolation Color Pixel Signals from a Subsampled Color 
Image," by Acharya et aL, filed on October 1 , 1999, assigned to the assignee of the present 
invention 

and herein incorporated by reference. 

10 

BACKGROUND 

This disclosure is related to interpolating color pixel signal values for an image and, in 
particular, interpolating color pixel signal values using hue. 

15 

As is well-known, in a variety of circumstances, it is desirable to perform color 
interpolation. For example, for a camera or other imager that has the capability of creating a 
color image, typically it is too expensive to have the capability to capture three independent 
color pixel signal values for each pixel location of an image at least on part because this 
20 would employ additional sensors. Therefore, more typically, a subsampled color image is 
captured and then the missing color pixel signal values are computed using techniques of 
color interpolation. Most of the existing simple color interpolation techniques typically do not 
produce high-quality color images. Therefore, a need continues to exist for color interpolation 
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techniques capable of producing quality color images from a subsampled color image. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The subject matter regarded as the invention is particularly pointed out and distinctly 
claimed in the concluding portion of the specification. The invention, however, both as to 
organization and method of operation, together with objects, features, and advantages 
thereof, may best be understood by reference to the following detailed description when read 
with the accompanying drawings in which: 

FIG. 1 is a schematic diagram illustrating a subsampled color image, such as a Bayer 
pattern color mage; 

FIG. 2 is a schematic diagram illustrating a portion of a subsampled color image in 
which the color pixel signal values are to be employed by an embodiment of a method in 
accordance with the present invention; 

FIG. 3 is a schematic diagram illustrating a portion of another subsampled color image 
in which the color pixel signal values are to be employed by an embodiment of a method in 
accordance with the present invention; 

FIG. 4 is a schematic diagram illustrating yet another portion of a subsampled color 
image in which the color pixel signal values are to be employed by an embodiment of a 
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method in accordance with the present invention. 
DETAILED DESCRIPTION 

5 In the following detailed description, numerous specific details are set forth in order to 

provide a thorough understanding of the invention. However, it will be understood by those 
skilled in the art that the present invention may be practiced without these specific details. In 
other instances, well-known methods, procedures, components and circuits have not been 
described in detail so as not to obscure the present invention. 

ft: As previously indicated, in a variety of circumstances it may be desirable to have a 

£ capability to interpolate color pixel signals from a subsampled color image in order to recover 
ffi the missing color components in a particular pixel location. FIG. 1 is a schematic diagram 
1^ illustrating a Bayer pattern color image, which is one example of a subsampled color image, 
15 although the invention is not limited in scope in this respect. For example, many other well- 
p known subsampled color images may be employed and will provide satisfactory results. As 
previously indicated, unfortunately, many color interpolation techniques typically do not 
produce high-quality color images at least in part because the techniques employed typically 
do not take into account, or at least reasonably correctly take into account, how the human 
20 eye perceives color. For example, a typical color interpolation technique may include 

averaging the pixels adjacent to a particular pixel location in which it is desired to interpolate 
the color signal value for those colors not included in that pixel location of a subsampled color 
image. However, typically, simply employing a pure average does not produce high quality 
results at least in part because it may blur the edge information and produce a color bleeding 
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problem, for example. 

In this particular embodiment of a method of using hue to interpolate color pixel signals 
in accordance with the present invention, color interpolation is accomplished by comparing, 
for a particular pixel location in a subsampled color image, relative differences in hue for two 
mutually orthogonal directions across the particular pixel location. Hue, in this context, for red 
and blue, refers to the relative signal value of a blue or red signal value to a green signal 
value at a given pixel location. It is assumed that if a significant change in hue occurs in a 
particular direction, the color pixel signal values in that direction are not as representative of 
the color pixel signal values at the particular pixel location as are the color pixel signal values 
from the mutually orthogonal direction. For example, it may be that in the image, an edge is 
being encountered. Therefore, in computing the color signal value for that particular pixel 
location, the change in hue for a direction that indicates less change in hue is weighed more 
heavily, in this particular embodiment. It is believed that this is more representative of how 
the human eye perceives color because the human vision system is more responsive to the 
relative change of color in neighboring pixels than to the color information in the pixel itself. In 
addition, as explained in more detail hereinafter, in computing pixel signal values for particular 
colors, the change in hue from both directions are weighed, instead of only using the change 
in hue from one direction to the exclusion of the other direction. 

FIG. 2 is schematic diagram illustrating a portion of a five pixel-by-five pixel grid of the 
Bayer pattern of FIG. 1. As illustrated, the center pixel location comprises a red pixel signal 
value. Therefore, for this particular pixel location, it is desired to interpolate the missing color 
signal values, blue and green, for that pixel location from the surrounding pixel signal values. 

HSJVIF/0367.doc-6/19/98-REV. 1-as 

5 



042390. P8746 

For example, a green pixel signal value may be estimated as follows. First, the relative 
change in hue for the horizontal direction and the vertical directions with respect to this 
particular pixel location, designated below as RCHJior and RCH_ver, respectively, may be 
computed and compared. This is accomplished using the following equations. 
RCHJior = 1 1 /4 x [- R m , n . 2 + 2 G m , n -i - 2 G m ,„ + i + R m , n+2 ] I 

[1] 

RCH_ver= \Y 2 x[-R m . 2 , n +2 Gm-1,n " 2 G m +1,n + Rm+2,n] | 

it is noted that for convenience, for the equations throughout the text of this specification, such 
as equations [1] above, it is assumed that the pixel signal values employed, such as Ry or Gy, 
for example, are the logarithm of the pixel signal values having the corresponding designation 
from the subsampled image. 

If the relative change in hue in the vertical direction is greater than the relative change 
in hue in the horizontal direction, then the values in the horizontal direction, that is, in this 
embodiment the green and red pixel signal values that are the immediately adjacent pixel 
signal values in the horizontal direction, are weighed more heavily. In this embodiment, the 
weight assigned to these, 0.5 here, has been chosen based at least in part on fuzzy 
assignment concepts and experimentation, although the invention is not limited in scope in 
this respect. It is noted that other weights may be employed and provide satisfactory results. 
At the same time, the weights assigned to vertical signal values have been chosen as 0.1 , 
although the invention is not limited in scope in this respect. On the basis of the above 
discussion, the missing green pixel signal values in this particular pixel location is estimated, 
in this embodiment, as 
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G m , n =[ 0.5* (Ihor) + 0.1 *(lver)]/(0.5 +0.1); or 



G m ,„ = 2[0.41667*(lhor) + 0.08333 * (Iver)]; 



5 



where 



[2] 



Ihor = (G m -i. n + G m+ i, n )/2 + 0.5 x (-R m . 2 , 



!,n + 2 R m , n " Rm+2,n)/4 



Iver = (R m , n .i + R m ,n-i)/2 + 0.5 x (-R m , n . 2 



+ 2 R m>n " Rm,n+2)/4 



K io If the relative change in hue in the horizontal direction is greater than the relative change in 
ri hue in the vertical direction, then a reverse approach is employed. More particularly, the pixel 

* signal signal values in the alternate directions, in this particular embodiment, are weighed 

o 

R more heavily. In particular, the green pixel signal value in this particular pixel signal location 
for such a situation is estimated as follows. 



It is noted that the form of this equation is similar to the form above, except that the vertical 
and horizontal pixel signal values that were computed are interchanged. Finally, if the two 
20 relative changes are equal, or substantially equal, then a simple average of Iver and Ihor are 
employed in this embodiment. 

G m ,n = 0.5 * (Ihor + Iver); 
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Therefore, in order to compute the signal value for the green color plane, where the particular 
pixel location has a pixel signal value in the red color plane, the relative change in hue in 
mutually orthogonal directions is compared. It is noted that where the particular pixel location 
has a blue pixel signal value in the Bayer pattern shown in FIG. 1 , the structure of a five pixel- 
5 by-five pixel portion of the sub-sampled color image is similar in arrangement to that illustrated 
in FIG. 2, except the center pixel contains blue signal information instead of red signal 
information. Therefore, for a pixel location containing a blue signal value, in this embodiment, 
the missing green pixel signal value for that pixel location may be estimated by using a 
technique essentially the same as one previously described, replacing the red pixel signal 
_ 10 value used in the equations above for a particular location with the blue pixel signal value for 
the particular location now being considered. 

€1 

m FIG. 3 is schematic diagram of another portion of the Bayer pattern of FIG. 1 . This 

s three pixel-by-three pixel portion is similar to a three-by-three portion of FIG. 2, except that 
f* is pixel signal values in the respective corners of the three-by-three array have also been 
}5 provided. For this portion, it is desirable to interpolate a blue pixel signal value for the 

location in which a red pixel signal value is provided. However, as illustrated in FIG. 3, blue 
pixel signal values immediately adjacent to the red pixel signal value are provided in the 
corners. Therefore, in contrast with the approach just described, the two mutually orthogonal 
20 directions to be compared comprise the main diagonal and the secondary diagonal of the 
array of pixels. Therefore, the relative change in hue across these two diagonals shall be 
compared. Hue, in this context, for red and blue, refers to the relative signal value of a blue or 
red signal value to a green signal value at a given pixel location. As previously indicated, the 
reason this approach is employed is because it is believed that hue information more 
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accurately reflects how the human eye perceives changes in color. Furthermore, a 
comparison for red and blue pixel signal values is made with green pixel signal values 
because, in Red-Green-Blue (RGB) color space format, the green plane includes a relatively 
larger amount of luminosity signal information compared to the red and blue planes. Hue for 
red or blue signal values, therefore, is estimated by comparison with the green value. 
Therefore, hue for the four blue pixel signal values is computed by comparing the blue pixel 
signal value in that location with the green pixel signal value in that location. It should be 
noted that the missing green pixel value has already been or may be recovered via the 
previous technique. Therefore, the following equations apply. 

hue_nw = B m .i, n -i - G m -i, n -i; 
hue_sw = B^.n.1 -G m+ i, n -i; 

[3] 

hue_ne = B m -i, n +i - G m-i,n+i; 

hlie_Se = B m +1,n+1 - G m+1,n+i; 

Then, the related change in hue along the main diagonal and the secondary diagonal may be 
computed as follows. 

huejnd = | huejiw - hue_se |; 

[4] 

hue_sd = | huejie - hue_sw |; 
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As discussed above, if the hue in any particular direction changes more significantly, then the 
hue in the mutually orthogonal direction is employed to estimate the blue pixel signal value in 
a particular pixel location. More accurately, that direction in which there is less relative 
change in hue is weighed more heavily than in the direction in which the relative change in 
hue is greater. Employing the relative change in hue in this fashion, the blue pixel signal 
value may be estimated as follows, assuming the relative relative change in hue along the 
main diagonal is less. 

B min = (G m , n ) + 0.41667 * ((B m -i f n-i ~ G m -1,n-l) + (B m +1,n+1 " G m +1,n+l)) + 

[5] 

0.08333 * ((B m -1,n+1 *G m-1,n+l) + (B m +1,n-1 - G m +1,n-l))i 

Likewise, a similar equation is employed for the opposing diagonal, except that the signal 
values along the diagonals are interchanged, providing the following equation. 

(B m .„) =(G m ,„) + 0.08333 *((B m-1,n-1 " G m-1,n-l) + (B m+1,n+1 "G m+1 ( nr+l)) + 

[6] 

0.41667 * ((B m -i, n+ i - G m-1,n+l) + (B m*1,n-1 - G m+ i, n -l)) 

It is noted that interpolation of a red pixel signal value for a particular pixel location 
having a blue pixel signal value may be obtained using a similar approach except that in the 
above equation the blue pixel signal values are interchanged with red pixel signal values. 
This is because, as illustrated in FIG. 1, where the particular pixel location has a blue pixel 

HS_MF/0367.doc-6/19/98-REV. 1-as 

10 



042390.P8746 

signal value, the pixel signal values in the four corners are red pixel signal values, providing 
substantially the same structure or array layout as illustrated in FIG. 3. Finally, a blue pixel 
signal value may be interpolated where a green pixel signal value is provided in the particular 
pixel location of the subsampled image, as illustrated in FIG. 4, for example. The approach is 
similar to that described in connection to the approach described with FIG. 3, except that, 
instead of employing the pixel signal values along the diagonals, the horizontal and vertical 
pixel signal values are employed using either horizontal or vertical pixel signal value directly 
available from the subsampled color image. The other pixel value in the orthogonal direction 
which is used for interpolation is obtained by employing the previous calculation. For 
example, the blue pixel signal values in the horizontal locations immediately adjacent the 
green pixel signal values in FIG. 4 are interpolated using the approach just described above 
in connection with FIG. 3. Therefore, the following equations are employed. 

hue_n = B m -i, n - G m -i, n ; 
hue_e = B m ,n+i - G m .n+i; 

[7] 

hue_W = B m ,n-l - G m,n-1 I 

hue_s - B m+ i, n - G m+ i, n ; 

hhor = hue_e - hue_w; 

hver = hue_n - hue_s, 
where B m , n+ i and B m ,„.i are obtained from the previous interpolation calculation. 
if I Ihhorl < Ihverl ) then 

(B m , n ) = (G m . n ) + 0.41667 * ((B m ^ - G m.n-1 ) + (B m ,n+i - G m ,„+l)) + 
HS_MF/0367.doc-6/19/98-REV. 1-as 

11 



042390. P8746 



0.08333 * ((B 



m- 



h1,n ~ G m-1,n) + (B m +1,n " G m +1,r>))- 



else 



(B m ,„) =(G m ^) + 0.08333 *((B m,n-1 " G m,n-l) (B m,n+l " G 



m,n+1 



0.41667 *((B m-1,n " G m-1,n) + (B m +l,n " G m+1,n)) 



endif 



A similar approach may be employed to interpolate the red pixel signal value where the 

particular pixel location of the subsampled image includes a green pixel signal value, except 
that in the equations provided above, the blue pixel signal values are replaced by red pixel 
signal values, as follows 



hue_n 


= R 


m-1 ,n 


-G 


m-1,n 


hue_e 


= R 


m,n+1 


-G 


m,n+1 


hue_w 


= R 


m,n-1 


-G 


m,n-1 1 



hhor = hue_e - hue_w ; 
hver = hueji - hue_s, 

where R m . 1>n and R m+1 , n (or R m ^ and R m , n+1 depending on whether the particular pixel is 
located in the red-green row or blue-green row) are obtained from the previous interpolation 
calculation. 



hue s 




/f( I hhorl<lhverl H/?en 
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(Rm,n) =(G m , n ) + 0.41667 *((R m,n-1 ~ G m,n-1 ) + (R 

m,n+1 " G m,n+l)) + 
0.08333 * ((R m-1.n " G m-1,n) + (R m+1,n " G m+1,n))- 

else 

(R m , n ) =(G m , n ) + 0.08333 *((R m,n-1 " G m,n-1 ) 

+ (R 

m,n+1 " G m,n+l)) + 

0.41667 * ((R m - 1>n - G m - 1in ) + (R m+1 , n - G m+1 , n )) 

endif 



it will, of course, be appreciated that the invention is not restricted in scope to a 
particular embodiment or implementation. For example, the foregoing approach, as one 
example of an approach in accordance with the invention, may be implemented in hardware, 
in software, in firmware, and/or any combination thereof. Again, intended merely as examples 
that do not limit the scope of the invention, an embodiment may comprise an imager including 
hardware, such as integrated circuit chips, that implement the foregoing. Alternatively, the 
imager may be coupled to a computing platform that includes software capable of 
implementing the foregoing. Likewise, a digital camera coupled to a desktop personal 
computer, for example, may implement an embodiment. Furthermore, these implementations 
in hardware and software may, of course, deviate from the foregoing and still be within the 
scope of the present invention. 

For embodiments that are at least partially implemented in software, such as, for 
example, the previously described embodiment, such software may reside on a storage 
medium, such as, for example, random access memory, a CD ROM, a floppy disk, or a hard 
drive, such that instructions are stored, which, when executed, such as by a computing 
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platform, such as a PC or other computing device, so that the system is capable of executing 
the instructions to result in the interpolation of color pixel signal values from a subsampled 
image. Likewise, such software may reside in firmware also, such as in flash memory or 
EEPROM, for example. 

While certain features of the invention have been illustrated and described herein, 
many modifications, substitutions, changes and equivalents will now occur to those skilled in 
the art. It is, therefore, to be understood that the appended claims are intended to cover all 
such modifications and changes as fall within the true spirit of the invention. 
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Claims : 

1 1 . A method of using hue to interpolate color pixel signal values comprising: 

2 for a particular pixel location in a subsampled image, comparing relative changes in 

3 hue for two mutually orthogonal directions across said particular pixel location; and 

4 computing a color signal value for that particular pixel location for a color plane other 

5 than the color plane of the pixel signal value in the subsampled color image at that location, 

6 the computation including relatively weighing the relative changes in hue, the relative weights 

7 depending, at least in part, on the difference in hue value in one particular direction relative to 

8 the other. 

ct 

n i 2. The method of claim 1 , wherein computing a color signal includes relatively weighing 

iTsf"? 

m 2 the differences in hue by relatively weighing more heavily the difference in hue associated 

* 3 with the direction having a difference in hue less relatively for the particular pixel location. 

s jit jj 

a i 3. The method of claim 2, wherein the subsampled image comprises an image in RGB 

™" 2 color space format. 

1 4. The method of claim 3, wherein the subsampled color image comprises a Bayer 

2 pattern. 

1 5. The method of claim 4, wherein the color plane of the pixel signal value at said 

2 particular pixel location comprises the R color plane; 

3 the two mutually orthogonal directions comprising the horizontal and vertical directions; 
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4 the particular color plane for the color signal value being computed comprising the G 

5 plane. 

1 6. The method of claim 4, wherein the color plane of the pixel signal value at said 

2 particular pixel location comprises the B color plane; 

3 the two mutually orthogonal directions comprising the horizontal and vertical direction; 

4 and 

5 the particular color plane for the color signal value being computed comprising the G 

6 plane. 

l 7. The method of claim 4, wherein the color plane of the pixel signal value at said 

2 2 particular pixel location comprises the R color plane; 

pl 3 the two mutually orthogonal direction comprising the main diagonal and the secondary 

^ 4 diagonal directions; 

j\ 5 the particular color plane for the color signal value being computed comprising the B 

K 6 plane. 

1 8. The method of claim 7, wherein the interpolation of a blue pixel signal value at a green 

2 pixel location is based at least in part on computed B pixel signal value levels for red pixel 

3 locations adjacent said green pixel location and also on existing blue pixel locations adjacent 

4 said green pixel location in a mutually orthogonal direction to said adjacent red pixel locations 

5 in the subsampled color image. 

l 9. The method of claim 4, wherein the color plane of the pixel signal value at said 
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2 particular pixel location comprises the B color plane; 

3 the two mutually orthogonal directions comprising the main diagonal and the secondary 

4 diagonal directions; 

5 the particular color plane for the color signal value being computed comprising the R 

6 plane. 

1 10. The method of claim 9, wherein the interpolation of a red pixel signal value at a green 

2 pixel location is based at least in part on computed R pixel signal value levels for blue pixel 

3 locations adjacent said green pixel location and also on existing red pixel locations adjacent 

4 said green pixel location in a mutually orthogonal direction to said adjacent blue pixel 

L..J 

^ 5 locations in the subsampled color image. 

yo 

03 

Jfi i 11. The method of claim 4, wherein the color plane of the pixel signal value at said 

^ 2 particular pixel location comprises the G color plane; 

*f] 3 the two mutually orthogonal direction comprising the horizontal and the vertical 

J5 4 directions; 

o 

5 the particular color plane for the color signal value being computed comprising the B 

6 plane. 

1 12. The method of claim 4, wherein the color plane of the pixel signal value at said 

2 particular pixel location comprises the G color plane; 

3 the two mutually orthogonal direction comprising the horizontal and the vertical 

4 directions; 

5 the particular color plane for the color signal value being computed comprising the R 
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6 plane. 

1 13. An article comprising: 

2 a storage medium, having stored thereon instructions, which, when executed by a 

3 system capable of executing the instructions, result in interpolating color pixel signal values 

4 from a subsampled image by: 

5 for a particular pixel location in the subsampled image, comparing changes in hue for 

6 two mutually orthogonal directions across said particular pixel location; and 

7 computing a color signal value for that particular pixel location for a color plane other 
^ 8 than the color plane of the pixel signal value in the subsampled color image at that location by 

fi 9 relatively weighing the differences in hue, the relative weights depending, at least in part, on 

if! 

y, io the difference in hue in a particular direction relative to the other direction. 

OQ 

!«_ ! 

^ i 14. The article of claim 13, wherein the instructions, when executed, further result in 

rv 2 interpolating color pixel signal values from a subsampled image in RBG color space format. 

1 1 5. The article of claim 1 3, wherein the instructions, when executed, further result in 

2 interpolating color pixel signal values from a Bayer pattern image. 

1 16. An integrated circuit comprising; 

2 electronic circuitry adapted to process pixel signal values; 

3 wherein said electronic circuitry is further adapted to interpolate color pixel signal 

4 values from a subsampled image by: 

5 for a particular pixel location in the subsampled image, comparing changes in hue for 
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two mutually orthogonal directions across said particular pixel location; and 

computing a color signal value for that particular pixel location for a color plane other 
than the color plane of the pixel signal value in the subsampled color image at that location by 
relatively weighing the differences in hue, the relative weights depending, at least in part, on 
the difference hue in a particular direction relative to the other direction. 

17. The integrated circuit of claim 16, wherein said electronic circuitry is further adapted to 
interpolate color pixel signal values from a subsampled image in RBG color space format. 

1 8. The integrated circuit of claim 16, wherein said electronic circuitry is further adapted to 
interpolate color pixel signal values from a Bayer pattern image. 

19. A system comprising; 

a computing platform adapted to process pixel signal values; 
wherein said computing platfrom is further adapted to interpolate color pixel signal values from 
a subsampled image by: 

for a particular pixel location in the subsampled image, comparing differences in hue 
for two mutually orthogonal directions across said particular pixel location; and 

computing a color signal value for that particular pixel location for a color plane other 
than the color plane of the pixel signal value in the subsampled color image at that location by 
relatively weighing the difference in hue, the relative weights depending, at least in part, on 
the difference in hue in a particular direction relative to the other direction. 

20. The system of claim 19, wherein said computing platfrom is further adapted to 
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interpolate color pixel signal values from a subsampled image in RBG color space format. 
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ABSTRACT 

Briefly, in accordance with one embodiment of the invention, a method of using hue to 
interpolate color pixel signals includes the following. For a particular pixel location in a 
subsampled color image, differences in hue are compared for two mutually orthogonal 
directions across the particular pixel location. A color signal value for that particular pixel 
location for a color plane other than the color plane of the pixel signal value in the 
subsampled color image of that location is computed. The computation includes relatively 
weighing the differences in hue values, the relative weights depending, at least in part, on the 
difference in hue value in one direction relative to the other. 
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Figure 1: Bayer Pattern 
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